Deep-Learning-Based Whole-Lung and Lung-Lesion Quantification Despite Inconsistent Ground Truth: Application to Computerized Tomography in SARS-CoV-2 Nonhuman Primate Models

Rationale and Objectives Animal modeling of infectious diseases such as coronavirus disease 2019 (COVID-19) is important for exploration of natural history, understanding of pathogenesis, and evaluation of countermeasures. Preclinical studies enable rigorous control of experimental conditions as well as pre-exposure baseline and longitudinal measurements, including medical imaging, that are often unavailable in the clinical research setting. Computerized tomography (CT) imaging provides important diagnostic, prognostic, and disease characterization to clinicians and clinical researchers. In that context, automated deep-learning systems for the analysis of CT imaging have been broadly proposed, but their practical utility has been limited. Manual outlining of the ground truth (i.e., lung-lesions) requires accurate distinctions between abnormal and normal tissues that often have vague boundaries and is subject to reader heterogeneity in interpretation. Indeed, this subjectivity is demonstrated as wide inconsistency in manual outlines among experts and from the same expert [1]. The application of deep-learning data-science tools has been less well-evaluated in the preclinical setting, including in nonhuman primate (NHP) models of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection/COVID-19, in which the translation of human-derived deep-learning tools is challenging. The automated segmentation of the whole lung and lung lesions provides a potentially standardized and automated method to detect and quantify disease. Materials and Methods We used deep-learning-based quantification of the whole lung and lung lesions on CT scans of NHPs exposed to SARS-CoV-2. We proposed a novel multi-model ensemble technique to address the inconsistency in the ground truths for deep-learning-based automated segmentation of the whole lung and lung lesions. Multiple models were obtained by training the convolutional neural network (CNN) on different subsets of the training data instead of having a single model using the entire training dataset. Moreover, we employed a feature pyramid network (FPN), a CNN that provides predictions at different resolution levels, enabling the network to predict objects with wide size variations. Results We achieved an average of 99.4 and 60.2% Dice coefficients for whole-lung and lung-lesion segmentation, respectively. The proposed multi-model FPN outperformed well-accepted methods U-Net (50.5%), V-Net (54.5%), and Inception (53.4%) for the challenging lesion-segmentation task. We show the application of segmentation outputs for longitudinal quantification of lung disease in SARS-CoV-2-exposed and mock-exposed NHPs. Conclusion Deep-learning methods should be optimally characterized for and targeted specifically to preclinical research needs in terms of impact, automation, and dynamic quantification independently from purely clinical applications.

Deep-Learning-Based Whole-Lung and Lung-Lesion Quantification Despite Inconsistent Ground Truth: Application to Computerized Tomography in SARS-CoV-2 Nonhuman Primate Models.

RATIONALE AND OBJECTIVES: Animal modeling of infectious diseases such as coronavirus disease 2019 (COVID-19) is important for exploration of natural history, understanding of pathogenesis, and evaluation of countermeasures. Preclinical studies enable rigorous control of experimental conditions as well as pre-exposure baseline and longitudinal measurements, including medical imaging, that are often unavailable in the clinical research setting. Computerized tomography (CT) imaging provides important diagnostic, prognostic, and disease characterization to clinicians and clinical researchers. In that context, automated deep-learning systems for the analysis of CT imaging have been broadly proposed, but their practical utility has been limited. Manual outlining of the ground truth (i.e., lung-lesions) requires accurate distinctions between abnormal and normal tissues that often have vague boundaries and is subject to reader heterogeneity in interpretation. Indeed, this subjectivity is demonstrated as wide inconsistency in manual outlines among experts and from the same expert. The application of deep-learning data-science tools has been less well-evaluated in the preclinical setting, including in nonhuman primate (NHP) models of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection/COVID-19, in which the translation of human-derived deep-learning tools is challenging. The automated segmentation of the whole lung and lung lesions provides a potentially standardized and automated method to detect and quantify disease. MATERIALS AND METHODS: We used deep-learning-based quantification of the whole lung and lung lesions on CT scans of NHPs exposed to SARS-CoV-2. We proposed a novel multi-model ensemble technique to address the inconsistency in the ground truths for deep-learning-based automated segmentation of the whole lung and lung lesions. Multiple models were obtained by training the convolutional neural network (CNN) on different subsets of the training data instead of having a single model using the entire training dataset. Moreover, we employed a feature pyramid network (FPN), a CNN that provides predictions at different resolution levels, enabling the network to predict objects with wide size variations. RESULTS: We achieved an average of 99.4 and 60.2% Dice coefficients for whole-lung and lung-lesion segmentation, respectively. The proposed multi-model FPN outperformed well-accepted methods U-Net (50.5%), V-Net (54.5%), and Inception (53.4%) for the challenging lesion-segmentation task. We show the application of segmentation outputs for longitudinal quantification of lung disease in SARS-CoV-2-exposed and mock-exposed NHPs. CONCLUSION: Deep-learning methods should be optimally characterized for and targeted specifically to preclinical research needs in terms of impact, automation, and dynamic quantification independently from purely clinical applications.

Toward the determination of sensitive and reliable whole-lung computed tomography features for robust standard radiomics and delta-radiomics analysis in a nonhuman primate model of coronavirus disease 2019.

Purpose: We propose a method to identify sensitive and reliable whole-lung radiomic features from computed tomography (CT) images in a nonhuman primate model of coronavirus disease 2019 (COVID-19). Criteria used for feature selection in this method may improve the performance and robustness of predictive models. Approach: Fourteen crab-eating macaques were assigned to two experimental groups and exposed to either severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) or a mock inoculum. High-resolution CT scans were acquired before exposure and on several post-exposure days. Lung volumes were segmented using a deep-learning methodology, and radiomic features were extracted from the original image. The reliability of each feature was assessed by the intraclass correlation coefficient (ICC) using the mock-exposed group data. The sensitivity of each feature was assessed using the virus-exposed group data by defining a factor R that estimates the excess of variation above the maximum normal variation computed in the mock-exposed group. R and ICC were used to rank features and identify non-sensitive and unstable features. Results: Out of 111 radiomic features, 43% had excellent reliability ( ICC > 0.90 ), and 55% had either good ( ICC > 0.75 ) or moderate ( ICC > 0.50 ) reliability. Nineteen features were not sensitive to the radiological manifestations of SARS-CoV-2 exposure. The sensitivity of features showed patterns that suggested a correlation with the radiological manifestations. Conclusions: Features were quantified and ranked based on their sensitivity and reliability. Features to be excluded to create more robust models were identified. Applicability to similar viral pneumonia studies is also possible.

Toward the determination of sensitive and reliable whole-lung computed tomography features for robust standard radiomics and delta-radiomics analysis in a nonhuman primate model of coronavirus disease 2019

. Purpose We propose a method to identify sensitive and reliable whole-lung radiomic features from computed tomography (CT) images in a nonhuman primate model of coronavirus disease 2019 (COVID-19). Criteria used for feature selection in this method may improve the performance and robustness of predictive models. Approach Fourteen crab-eating macaques were assigned to two experimental groups and exposed to either severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) or a mock inoculum. High-resolution CT scans were acquired before exposure and on several post-exposure days. Lung volumes were segmented using a deep-learning methodology, and radiomic features were extracted from the original image. The reliability of each feature was assessed by the intraclass correlation coefficient (ICC) using the mock-exposed group data. The sensitivity of each feature was assessed using the virus-exposed group data by defining a factor R that estimates the excess of variation above the maximum normal variation computed in the mock-exposed group. R and ICC were used to rank features and identify non-sensitive and unstable features. Results Out of 111 radiomic features, 43% had excellent reliability ( Conclusions Features were quantified and ranked based on their sensitivity and reliability. Features to be excluded to create more robust models were identified. Applicability to similar viral pneumonia studies is also possible.

Medical imaging of pulmonary disease in SARS-CoV-2-exposed non-human primates.

Chest X-ray (CXR), computed tomography (CT), and positron emission tomography-computed tomography (PET-CT) are noninvasive imaging techniques widely used in human and veterinary pulmonary research and medicine. These techniques have recently been applied in studies of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)-exposed non-human primates (NHPs) to complement virological assessments with meaningful translational readouts of lung disease. Our review of the literature indicates that medical imaging of SARS-CoV-2-exposed NHPs enables high-resolution qualitative and quantitative characterization of disease otherwise clinically invisible and potentially provides user-independent and unbiased evaluation of medical countermeasures (MCMs). However, we also found high variability in image acquisition and analysis protocols among studies. These findings uncover an urgent need to improve standardization and ensure direct comparability across studies.

Animal models for COVID-19.

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the aetiological agent of coronavirus disease 2019 (COVID-19), an emerging respiratory infection caused by the introduction of a novel coronavirus into humans late in 2019 (first detected in Hubei province, China). As of 18 September 2020, SARS-CoV-2 has spread to 215 countries, has infected more than 30 million people and has caused more than 950,000 deaths. As humans do not have pre-existing immunity to SARS-CoV-2, there is an urgent need to develop therapeutic agents and vaccines to mitigate the current pandemic and to prevent the re-emergence of COVID-19. In February 2020, the World Health Organization (WHO) assembled an international panel to develop animal models for COVID-19 to accelerate the testing of vaccines and therapeutic agents. Here we summarize the findings to date and provides relevant information for preclinical testing of vaccine candidates and therapeutic agents for COVID-19.